A computational model for the influence of cross modal context upon syntactic parsing

Ambiguity is an inherent property of natural language. Its most prominent manifestations comprise syntactic ambiguity, lexical ambiguity, scope ambiguity and referential ambiguity. Considering the high frequency with which ambiguity occurs in unrestricted natural language, it is surprising how seldom ambiguity causes misunderstandings. Most linguistic ambiguities in inter-human communication even pass unnoticed, mainly because human cognition automatically and unconsciously attempts to resolve ambiguity. A central contribution to this automatic and unconscious disambiguation is made by the integration of non-linguistic information from cognitively readily available sources such as world knowledge, discourse context or visual scene context. While a large body of behavioural investigations into the interactions between vision and language has been accumulated, comparatively few computational models of those interactions have been reported. The focus of this thesis is to motivate, specify and validate a computational model for the cross-modal influence of visual scene context upon natural language understanding and the process of syntactic parsing, in particular. We argue for a computational model that establishes cross-modal referential links between words in the linguistic input and entities in a visual scene context. Cross-modal referential links are assigned on the basis of conceptual compatibility between the concepts activated in the linguistic modality and the concepts instantiated in visual context. The proposed model utilises the thematic relations in the visual scene context to modulate attachments in the linguistic analysis. In contrast to the majority of extant computational models for the interaction between vision and language, our model is motivated by an integrated theory of cognition. We base our model architecture on the cognitive framework of Conceptual Semantics, an overarching theory of cognition and language processing by Ray Jackendoff. In our model, we adopt the central tennet of Conceptual Semantics that all cross-modal interactions of non-linguistic modalities with language are mediated by Conceptual Structure, a single, uniform representation of linguistic and non-linguistic semantics. Conceptual Structure propagates the influence of the nonlinguistic modalities into syntactic representation via a syntax-semantics interface. The purpose of this interface is to map between the syntactic and the semantic representation by means of representational correspondence rules. Our model implements central aspects of the cognitive architecture in Conceptual Semantics. We encode the semantic information for all entities, be they linguistic or non-linguistic in nature, on a single level of semantic representation. In particular, the semantic part of linguistic analysis and visual scene information are included in vii